E2B-1291 UnicodeDecodeError in Python SDK when using curl command

⚠️ No Changeset found

Latest commit: bff11dc

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

1. Can this happen in the JS too?
2. Aren't we missing the change is async?

1. Can this happen in the JS too?
2. Aren't we missing the change is async?

1. @jamesmurdza tested it with js-sdk, it doesn't throw, it uses replacement chars by default.
2. Adding it to async 👍

Do we know why it is happening? Can we improve the encoding in envd or are we doing something improper while encoding?

My question is basically—why are there undecodable chars in the stdout/stderr? Is this because of the command output or are we messing something up in the envd?

Do we know why it is happening? Can we improve the encoding in envd or are we doing something improper while encoding?

Essentially when reading data from a network socket, the client might encounter a scenario where a multi-byte UTF-8 character is split across network reads. UTF-8 uses multiple bytes to represent characters with codepoints above 127, and these characters can be interrupted mid-sequence when the buffer is filled.

Do we know why it is happening? Can we improve the encoding in envd or are we doing something improper while encoding?

Essentially when reading data from a network socket, the client might encounter a scenario where a multi-byte UTF-8 character is split across network reads. UTF-8 uses multiple bytes to represent characters with codepoints above 127, and these characters can be interrupted mid-sequence when the buffer is filled.

Should this happen here when the messages we are getting from envd are actually structured and contain multiple fields in addition to the stdout/stderr?

With that in mind, should we fix this when reading the messages here (

With that in mind, should we fix this when reading the messages here (

E2B/packages/python-sdk/e2b_connect/client.py

Line 395 in 5f022cc

for chunk in http_resp.iter_stream():

) because we might not be reading the whole message?

IIUC you don't get to control where the split happens when reading byte stream from the network, but at the same time im surprised this is not a solved problem given the ubiquity of utf8 encoded byte array streaming

With that in mind, should we fix this when reading the messages here (

E2B/packages/python-sdk/e2b_connect/client.py

Line 395 in 5f022cc

for chunk in http_resp.iter_stream():

) because we might not be reading the whole message?

That sounds like a separate PR. This PR looks good to me!